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DUPLICATE 

1 

Selecting Functions Via a Graphical User Interface 



Background of the Invention 

1 . Field of the Invention 

The present invention relates to apparatus for processing image data 
and a method of selecting a function via a graphical user interface. 

2. Description of the Related Art 

Systems for processing image data, having a processing unit, 
storage devices, a display device and a stylus-like manually operable input 
device (such as a stylus and touchtablet combination) are shown in United 
States Patents 5,892,506; 5,786,824 and 6,269,180 all assigned to the 
present Assignee. In these aforesaid systems, it is possible to perform 
many functions upon stored image data in response to an operator 
manually selecting a function from a function menu. 

Recently, in such systems as "FIRE" and "INFERNO", licensed by 
the present Assignee, the number of functions that may be performed have 
increased significantly. Thus, for example, there has been a tendency 
towards providing functions for special effects, compositing and editing on 
the same platform. 

Function selection is often done via graphical user interfaces in 
which menus are displayed from which a selection may be made. A 
function selection using a _ menu Js ach a cursor over to a 

selection position within the menu-by operation qf the stylus. The particular 



function concerned is selected by placing the stylus into pressure; an 
operation logically similar to a mouse click. Menus of this type are used in 
systems where stylus-like input devices are preferred, in preference to 
pulldown menus, given that, with pulldown menus, it is necessary to 
maintain stylus pressure while menu selection takes place. Such an 
operation places unnecessary strain on the wrists and fingers of an 
operator and is therefore not preferred in applications that make significant 
use of stylus-like devices. 

In addition to there being a trend towards increasing the level of 
functionality provided by digital image processing systems, there has also 
been a trend towards manipulating images of higher definition. Initially, 
many systems of this type were designed to manipulate standard broadcast 
television images such as NTSC or PAL. With images of this type, it is 
possible to display individual frames on a high definition monitor such that 
the displayed images take up a relatively small area of the monitor thereby 
leaving other areas of the monitor for displaying menus etc. Increasingly, 
digital techniques are being used on high definition video images or images 
scanned from cinematographic film. Such images have a significantly 
higher pixel definition. Consequently, even when relatively large monitors 
are. used, there may be very little additional area for the display of menus. 

Furthermore, operators and artists are under increasing pressure to 
increase the rate at which work is finished. Being able to work with systems 
of this type quickly and efficiently is not facilitated if complex menu 
structures are provided or manipulation tools are provided that are not 



intuitive to the way artists work. 

Brief Summary of the Invention 

According to a first aspect of the present invention, there is provided 
apparatus for processing image data, comprising processing means, 
storage means, display means and stylus-like manually operable input 
means, wherein said processing means is configured to perform functions 
upon image data in response to an operator manually selecting a function 
from a function menu; said processing means responds to a first user- 
generated input command so as to display a plurality of function gates at a 
cursor position; movement of the stylus-like manually operable input means 
so as to move said cursor through one of said function gates results in a 
related menu being displayed; and manual selection of a function from said 
displayed menu results in the selected function being performed upon said 
image data. 

Brief Description of the Several Views of the Drawings 

Figure 1 shows a system for processing image data that embodies 
the present invention; 

Figure 2 shows details the computer system shown in Figure 7; 

Figure 3 shows illustrates the display of the prior art; 

Figure 4 shows the display of Figure 3 with graphically displayed 
menus as is known in the prior art; 

Figure 5 shows an example of a scene graph defining how a 
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complex scene is rendered; 

Figure 6 is the monitor of Figure 1 displaying a high definition image; 

Figure 7 shows a portion of the image shown in Figure 6 with user 
interface gates embodying the present invention; 

Figure 8 shows an abstracted view of the gates shown in Figure 7; 

Figure 9 shows the high definition image of Figure 6 with an overlaid 
upper menu; 

Figure 10 shows the high definition image of Figure 6 with a lower 

menu; 

Figure 11 shows the high definition of Figure 6 with a menu to the 

left; 

Figure 12 shows the high definition image of Figure 6 with a menu to 
the right; 

Figure 13 identifies operations performed by the processing unit 

shown in Figure 2; 

Figure 14 details procedures identified in Figure 13] 
Figure 15 details procedures identified in Figure 14] 
Figure 16 details procedures identified in Figure 15; 
Figure 17 identifies a first alternative embodiment of the present 

invention; 

Figure 18 identifies further alternative embodiments of the present 
invention. 



Written Description of the Best Mode for Carrying Out the Invention 
Figure 1 

Preferred apparatus for processing image data and embodying the 
present invention is illustrated in Figure 1. A computer system 101 supplies 
output signals to a visual display unit 102. The visual display unit 102 
displays images, menus and a cursor and movement of said cursor is 
controlled in response to manual operation of a stylus 103 upon a touch 
table 104. In addition, input data is also supplied to the computer system 
101 via a keyboard 105. Keyboard 105 is of a standard alpha numeric 
layout and includes a spacebar 106. Manual operation of the spacebar 106 
provides a first input command in a preferred embodiment resulting in a 
quadrilateral device being displayed at the cursor position. The quadrilateral 
device identifies a function type at each of its four edges, each having an 
associated displayable menu. In response to a second input command, 
preferably received from the stylus 103, the cursor is moved over one of the 
edges of the displayed quadrilateral device. Thereafter, having moved the 
cursor over an edge of the quadrilateral device, the aforesaid menu 
associated with the edge over which the cursor has been moved is 
displayed. In this way, a user is given rapid access to a menu of interest 
without said menu being continually displayed over the working area of the 
VDU102. 



Figure 2 



Computer system 101 is illustrated in Figure 2. System bus 201 
provides communication between a central processing unit 202, random 
access storage devices 203, a video card 204, disk storage 205, CD ROM 
reader 206, a network card 207, a tablet interface card 208 and a keyboard 
interface card 209. Typically, the central processing unit may be an Intel 
based processor operating under the Windows operating system. Program 
instructions for the central processing unit 202 are read from the random 
access memory device at 203. Program instructions embodying the present 
invention are preferably received via a CD ROM 210 for installation within 
the storage system of disk drive 205 via the CD ROM reader 206. 

Network card 204 supplies output signals to monitor 102 with input 
signals from the tablet 104 being received via a tablet interface 208 and 
input signals from keyboard 105 being received via the keyboard interface 
209. Network interface 207 allows the system to exchange files with a 
server or other networked stations. 

Figure 3 

A monitor 301, similar of a prior art system and not that shown in 
Figure 1 is illustrated in Figure 3. The monitor is displaying a video image 
302 consisting of a plurality of frames played over a period of time at 
standard broadcast definition. The monitor has a substantially higher 
definition, thereby ensuring that there is plenty of space around the image 
302 for graphical interfaces to be displayed. 



Figure 4 

Monitor 301 is shown in Figure 4 with a plurality of menus, such as 
menu 304 and menu 305, displayed around video image 302. In this way, 
many control functions may be selected by appropriate operation of the 
stylus 103 upon a touch-tablet 104. A function of interest is selected by 
placing the cursor over a soft button. The button is then depressed by 
placing the stylus 103 into pressure. This may result in a function being 
performed upon the image directly or, alternatively, may result in an 
appropriate sub-menu being displayed so that appropriate control may be 
made in response to user input. 

Figure 5 

It can be appreciated that the working space displayed on monitor 
301 has become somewhat complex if all available functions are to be 
displayed. 

The number of possible functions available to an artist has increased 
but increasingly more and more of these functions are used concurrently to 
produce a particular effect. Furthermore, it is preferable for the nature of the 
functions to be stored as definitions or metadata whereafter their 
implementation takes place in real-time. Thus, the process of compositing 
etc requires many functions to be performed as part of a final rendering 
operation rather than partially processed work being stored and then 
processed upon again. Consequently, many functions may be required and 
in order to make modifications an artist is required to identify a particular 



8 

function of interest. 

In order to provide artists with a representation of the nature of a 
function being performed, the structure of the processing operations may 
be displayed as a process tree, as illustrated in Figure 5. The process trees 
generally consist of sequentially linked processing nodes, each of which 
specifies a particular processing task required in order to eventually 
achieve an output in the form of a composited frame or video sequence. 
Traditionally, an output sequence 501 will comprise both image data and 
audio data. Accordingly, the composited scene will require the output from 
an image keying node 502 and the output from a sound mixing node 503. 
In this example, the image keying node 502 calls on a plurality of further 
processing nodes to obtain all the input data it requires to generate the 
desired image data, or sequence of composited frames. In the example, the 
desired output image data includes a plurality of frames within which a 
three-dimensional computer generated object is composited, as well as a 
background also consisting of a plurality of three-dimensional objects 
superimposed over a background texture. 

The image keying node 502 initially requires a sequence of frames 
504, each frame of which is substantially processed by a colour correcting 
processing node 505 and a motion tracking processing node 506 such that 
the composited three-dimensional object generated by three-dimensional 
modelling node 507, to which is applied a texture by the texturing node 508 
and appropriate lighting by artificial light processing node 509 and finally 
appropriate scaling by scaling node 510 is seamlessly composited within 



the colour correcting sequence of frames 104. For the background, the 
image keying processing node 502 also requires a uniform texture from a 
texturing node 511, the functionality of which is similar to texturing node 
508, to which is applied the colour correction functionality of a colour 
correction processing node 512, the functionality of which is similar to the 
colour correcting processing node 505. The image keying processing node 
502 is finally required to overlay the plurality of simple three-dimensional 
objects generated from the three-dimensional modelling node 513, which 
are appropriately lit by the artificial light processing node 514 and motion 
tracked by motion tracking processing node 515 over the colour corrected 
texture 511 before overlaying a compositive frame sequence of node 504 
on top of the composited background. 

Each node illustrated in Figure 5 will have an associated menu of 
controls allowing modifications to be made at that particular point in the 
overall image processing exercise. Thus, when modifications are made at 
the menu level, it is necessary for a database to be established so as to 
oversee the relationship between manual input commands being made and 
their associated node at which the modifications are take effect. Thus, the 
complexity of images results in a greater requirement for the display of 
control menus so as to allow full control to be given to an artist during a 
compositing exercise. 

Figure 6 

Problems associated with the availability with monitor space are 
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made worse when the definition of images being processed is increased. 
Figure 3 shows a prior art example of a standard television broadcast 
image being processed. However, as illustrated in Figure 6, the present 
invention is particularly directed towards the processing of higher definition 
images such as images derived from cinematographic film. Thus, a high 
definition image has been loaded of a definition such that, when displayed, 
as illustrated in Figure 6, the whole of the available display space of visual 
display unit 102 is used for displaying the image frames. Even with very 
large visual display units, it is recognised that artists must work with 
material at an appropriate definition so as to ensure that the introduction of 
visible artefacts is minimised. However, a problem with displaying images at 
this definition as illustrated in Figure 6, is that the monitor does not provide 
additional space with a display of menus alongside the displayed high 
definition images. 

Region 602 of the high definition image 601 is shown enlarged in 
Figure 7. A cursor 603 is shown in Figure 6 at a selected position. After 
being placed in this selected position, an artist operates spacebar 106 of 
the keyboard 105 resulting in a quadrilateral device being displayed at the 
cursor position. 

Figure 7 

A quadrilateral graphical user interface device providing four regions 
that have been identified as "gates" is shown at 701 in Figure 7. Each gate 
of the quadrilateral device identifies a function type and each of said 
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function types has an associated displayable menu. For activating the 
spacebar, the quadrilateral device 701 is located around the position of the 
displayed cursor 602. The quadrilateral device 701 remains displayed while 
the spacebar 106 is held down by the artist. The device 701 may be 
removed simply by removing pressure from the spacebar 106. Moving the 
stylus 103 in an upwards direction results in the displayed cursor 602 
passing through the "viewer" gate 702. In response to passing the cursor 
through the viewer gate 702, a viewer menu is displayed in an upper 
portion of the screen. Similarly, by moving the stylus 103 in a downward 
direction, the cursor 602 is passed through tool control gate 703, identified' 
as the object tool in Figure 7. By moving the stylus 103 to the left, the 
cursor 602 passes through a "layer" gate 704 resulting in an associated 
menu being displayed to the left of the image. Furthermore, by moving the 
stylus 103 to the right, the displayed cursor 602 is taken through the tools 
gate 705, resulting in an appropriate menu being displayed to the right of 
the image. 

The particular function types available are relevant to the application 
being performed in the preferred embodiment. However, it should be 
appreciated that similar techniques may be used in different environments. 

Figure 8 

An abstracted interface is illustrated in Figure 8. In response to a first 
input command, a quadrilateral device 801 is displayed at a cursor position. 
In the preferred embodiment, this first input command consists of the 



12 

spacebar of a keyboard being depressed. The quadrilateral device 
identifies a function type at each of its four edges and by passing the cursor 
802 through one of these function types, an appropriate menu is displayed, 
preferably at a location related to the gate through which the cursor has 
been passed. Thus, if the cursor 802 moves to the left, preferably a left 
menu is displayed; if the cursor 802 moves to the right, preferably a right 
menu is displayed; if the cursor 802 moves upwards, preferably an upper 
menu is displayed; and if the cursor 802 moves downwards, preferably a 
lower menu is displayed. 

Figure 9 

Movement of cursor 602 in response to stylus 103 in an upwards 
direction through gate 702 results in a viewer gate menu 901 being 
displayed in an upper portion of the screen. The viewer gate menu is used 
to set viewer specific options such as render pre-sets for three-dimensional 
players or filtering for schematics. The viewer menu relates directly to the 
viewer in focus and the name of the viewer in focus preferably appears in 
the gate user interface. The displayed menu takes up the same width as a 
tool panel user interface and it is locked to the top of the user interface 
regardless of how many viewers are present. The panel is fully opaque and 
sits over all other panels. Upon leaving the viewer gate menu the menu 
itself disappears thereby returning the full screen to the image under 
consideration. 
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Figure 10 

Moving the cursor 602 in a downward direction, through gate 703, 
results in a current tool menu 901 being displayed in a lower region of the 
screen of monitor 102. The current tool menu is used to interact with the 
current tool. Gate 703 is only available if one tool has been selected. Thus, 
the gate relates directly to the current tool under consideration. The name 
of the current tool preferably appears in the gate user interface. The menu 
is locked to the bottom of the player in focus and use is also made of the 
transport tool user interface. 

After use has been made of the current tool menu, the menu is 
removed by activating spacebar 106 again, thereby making the whole 
screen available for the whole image. 

Figure 11 

Upon moving cursor 602 in a leftward direction through gate 704, a 
layer gate menu 1101 is displayed. The layer menu is used to select layers 
and the layer user interface takes up the same width as a layer list. It is 
locked to the left side of the user interface regardless of how many viewers 
are present. The panel is fully opaque and sits over all other panels. The 
layer gate menu 1101 only contains details of the layers; the layer list is not 
expandable and there is no value column. A user can set whether a layer is 
visible or not visible and the layer menu 1101 disappears after the cursor 
exits to a new area. 
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Figure 12 

Upon moving cursor 602 in a rightwards direction through gate 705 
tools menu 1201 is displayed. The tools menu is used to select the current 
tool and is only available when only one layer has been selected. The tools 
gate menu takes up the same width as the layer list and is locked to the 
right side of the interface regardless of how many viewers are present. The 
panel is fully opaque and sits over all other panels. The tools menu 1201 
contains a filtered version of the schematic showing only the tools 
associated with a selected object. The menu disappears after the cursor 
has been moved out of the menu area. 

Figure 13 

Operations performed by the processing unit 202 in order to provide 
the functionality described with reference to Figures 6 to 12, is identified in 
Figure 13. After power-up an operating system is loaded at step 1301 
whereafter at step 1302 the system responds to instructions from a user to 
run the compositing application. 

At step 1303 data files are loaded and at step 1304 the application 
operates in response to commands received from a user. At step 1305 
newiy created data is stored and at step 1306 a question is asked as to 
whether another job is to be processed. When answered in the affirmative, 
control is returned to step 1303 allowing new data files to be loaded. 
Alternatively, if the question asked at step 1306 is answered in the 
negative, the system is shutdown. 
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Figure 14 

Procedures 1304 relevant to the present preferred embodiment are 
illustrated in Figure 14. At step 1401 a keyboard operation is captured and 
at step 1402 a question is asked as to whether the spacebar has been 
activated. If answered in the negative, control is returned to step 1401 else 
control is directed to step 1403. 

In response to the spacebar being activated and detected at step 
1402, selection gates 701 are displayed at step 1403. At step 1404 a 
question is asked as to whether the spacebar has been released and if 
answered in the affirmative, the selection gates are removed. Alternatively, 
if the question asked at step 1401 is answered in the negative, control is 
directed to step 1406 such that the application responds to further cursor 
movement. 

Figure 15 

Procedure 1406 is detailed in Figure 15. At step 1501 cursor 
movement is captured and at step 1502 a question is asked as to whether 
the cursor has moved across the upper gate 702. If answered in the 
negative, control is directed to step 1505, but if answered in the affirmative 
the upper menu (the viewer menu in the preferred embodiment) is 
displayed at step 1503 and the system responds to menu selections made 
at step 1504. 

At step 1504 a question is asked as to whether the cursor has 
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crossed the lower gate 703 and if answered in the negative control is 
directed to step 1508. If answered in the affirmative, to the effect that the 
cursor did cross the lower gate 703, the lower gate menu (selected tool 
menu in the preferred embodiment) is displayed at step 1506 and 
responses to selections are made at step 1507. 

At step 1508 a question is asked as to whether the cursor has 
crossed the left gate 704 and if answered in the negative control is directed 
to step 705. In answered in the affirmative, the left gate menu (the layer 
menu in the preferred embodiment) is displayed at step 1509 and 
responses to selections are made at step 1510. 

At step 1511 a question is asked as to whether a cursor has crossed 
the right gate 705. If answered in the affirmative, the right gate menu (the 
tools menu in the preferred embodiment) is displayed at step 1512 and the 
system responds to manual selections at step 1513. 

Figure 16 

Procedures 1504 for responding to input selections are detailed in 
Figure 16. At step 1601 a position is captured when the stylus 103 is placed 
in pressure. 

At step 1602 a question is asked as to whether a menu has been 
closed, either as a result of a "close menu" button being operated or, for 
certain menus, whether the stylus has been taken outside the menu area. If 
answered in the affirmative, the menu is closed at step 1603. 

If the question asked at step 1602 is answered in the negative, a 
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question is asked at step 1604 as to whether a function has been selected. 
If answered in the affirmative, the function is called at step 1605. 

Procedures 1507, 1510 and 1513 are substantially similar to 
procedures 1504 shown in Figure 16. 

Figure 17 

An alternative embodiment is illustrated in Figure 17. In this 
embodiment, displayed quadrilateral devices representing gates are 
nested. On operation of the spacebar 106, a first device 1701 is displayed. 
Subsequent movement of the cursor to the left results in a further gate 
"Gate A" being selected as illustrated at 1702. Similarly, movement in an 
upwards direction results in a second gate "Gate B" being selected as 
illustrated at 1073. Similarly, movement to the right results in a further gate 
"Gate C" being selected as illustrated at 1704. Finally, movement in a 
downwards direction results in a further gate "Gate D" being selected as 
illustrated at 1705. 

Thus, movement of the cursor through any of the four gates shown 
in device 1701 results in a further gate being displayed, either Gate A, Gate 
B, Gate C or Gate D depending upon the direction of movement. Similarly 
movement then allows specific functions to be selected or, in an alternative 
embodiment, further nestings may be selected. Thus, in this example, upon 
producing Gate A, it is then possible to select functions F1, F2, F3 or F4. 
Similarly the presentation of Gate B allows further functions F5, F6, F7 or 
F8 to be selected. Similarly, the presentation of Gate C allows functions F9, 
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F10, F11 or F12 to be selected. Finally, the presentation of Gate D allows 
functions F13, F14, F15 or F16 to be selected. 

In a preferred embodiment, the displayed device is a quadrilateral 
and thereby allows four selections to be made. In example 1801 shown in 
5 Figure 18, only one of three possible selections needs to be made. Thus, it 
is possible to move a cursor to the left to select function F1, to move the 
cursor upwards to select function F2 or to move the cursor to the right to 
select function F3. No response is obtained if the cursor is moved in a 
downwards direction. 
10 An alternative approach to representing these three functions F1 , F2 

and F3 is illustrated at 1802. Here, as an alternative to being placed in a 
quadrilateral device, the device is substantially triangular. Similarly at 1803 
six selections may be made, functions F1 , F2, F3, F4, F5 or F6 by means of 
a substantially hexagonal device. 
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1. Apparatus for processing image data, comprising processing 
means, storage means, display means and stylus-like manually operable 
input means, wherein 

said processing means is configured to perform functions upon 
image data in response to an operator manually selecting a function from a 
function menu; 

said processing means responds to a first user-generated input 
command so as to display a plurality of function gates at a cursor position; 

movement of the stylus-like manually operable input means so as to 
move said cursor through one of said" function gates results in a related 
menu being displayed; and 

manual selection of a function from said displayed menu results in 
the selected function being performed upon said image data. 

2. Apparatus according to claim 1, wherein said manually 
operable input means is a stylus and a touch-tablet combination. 

3. Apparatus according to claim 1, wherein a first user- 
generated input command is generated in response to keyboard operation. 

4. Apparatus according claim 3, wherein said keyboard 
operation involves activation of a spacebar. 
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5. Apparatus according to claim 1, wherein four function gates 
define a substantially quadrilateral shape. 

6. Apparatus according to claim 1, wherein said menus relate to 
functions applicable to image data processing. 

7. Apparatus according to claim 6, wherein said image data 
processing functions relate to compositing and editing image frames. 

8. A method of selecting a function via a graphical user 
interface for receiving input commands, wherein 

in response to a first input command, a quadrilateral device is 
displayed at a cursor position; 

said quadrflateral device identifies a function type at each of its four 
edges, each having an associated displayable menu; 

in response to a second input command, a cursor is moved over one 
of said edges; and 

having moved the cursor over an edge or the quadrilateral device the 
aforesaid menu associated with the edge over, which the cursor has been 
moved is displayed. 

9. A method of supplying input data to a computer system, 
comprising the steps of 
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issuing a first input command to call up a graphical user interface in 
which a plurality of gates surround a cursor position; and 

in response to a second input command, moving said cursor through 
one of said gates. 

10. A method according to claim 9, wherein four gates are 
displayed in said graphical user interface in a configuration substantially 
quadrilateral. 

11. A method according to claim 9, wherein the passing through a 
gate of said graphical user interface results in a further lower level of gates 
being displayed. 

12. A computer-readable medium having computer-readable 
instructions executable by a computer such that, when executing said 
instructions, said computer will perform the steps of: 

responding to a first user-generated input command so as to display 
a plurality of function gates at a cursor position; 

responding to movement of manually operable input means so as to 
move said cursor through one of said function gates and displaying a menu 
in response to said cursor movement; and 

responding to manual selection of a function from said displayed 
menu so as to perform said function upon image data. 
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13. A computer-readable medium having computer-readable 
instructions according to claim 12, wherein said cursor moves through one 
of said function gates in response to manual operation of a stylus upon a 
touch-tablet. 

5 

14. A computer-readable medium having computer-readable 
instructions according to claim 12, such that when executing said 
instructions a computer will display four function gates that define a 
substantially quadrilateral shape. 

10 * 

15. A computer-readable medium having computer-readable 
instructions according to claim 12, such that when executing said 
instructions a computer will display a menu at a screen position related to 
the relative positions of its respective gate. 
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Abstract of the Disclosure 

Selecting Functions Via a Graphical User Interface 

A graphical user interface allows function commands to be selected, 
such as function commands applied to image data. A first user-generated 
input command, such as the pressing of a spacebar on a keyboard, 
displays a plurality of function gates (701) at a cursor position (602). 
Movement of a stylus or similar device through one of said displayed gates 
(702, 703, 704, 705) results in a related menu being displayed from which a 
specific function may be selected. 



(Figure 7) 
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